Skip to content

feat(extraction): add Groovy language support - #1

Closed
ymstar wants to merge 1 commit into
mainfrom
feat/groovy-support
Closed

ymstar wants to merge 1 commit into
mainfrom
feat/groovy-support

Conversation

@ymstar

@ymstar ymstar commented Jun 3, 2026

Copy link
Copy Markdown
Owner

Summary

Add tree-sitter based extraction for Groovy (.groovy, .gradle, .gvy, .gy, .gsh). Supports classes, methods, def functions, interfaces, enums, imports, fields, visibility modifiers, package declarations, and bare calls (juxt_function_call).

This is a rework of colbymchenry#302 with improvements based on the ADDING-A-LANGUAGE.md cookbook review.

Changes

File Change
src/types.ts Add 'groovy' to LANGUAGES
src/extraction/grammars.ts WASM mapping, vendored loading, extension map, display name
src/extraction/wasm/tree-sitter-groovy.wasm Vendored grammar (extends tree-sitter-java)
src/extraction/languages/groovy.ts New LanguageExtractor (Path A, based on Java)
src/extraction/languages/index.ts Register groovyExtractor
__tests__/extraction.test.ts 11 test cases

Test coverage

  • Language detection (5 extensions)
  • Class, method, function, interface, enum extraction
  • Import extraction
  • Visibility modifiers (public/private/protected)
  • Package declarations (cross-file resolution)
  • Bare calls (println "hello" via juxt_function_call)

Known limitation

Fields declared without an explicit type (e.g. private repository) are misparsed as method_declaration by the tree-sitter grammar — typed fields (e.g. private String name) extract correctly.

Test plan

  • npx tsc --noEmit — no errors
  • npx vitest run __tests__/extraction.test.ts -t "Groovy" — 11/11 passed
  • Full extraction test suite passes

Closes colbymchenry#169
Relates to colbymchenry#648
Supersedes colbymchenry#302

Add tree-sitter based extraction for Groovy (.groovy, .gradle, .gvy,
.gy, .gsh). Supports classes, methods, functions, interfaces, enums,
imports, fields, visibility modifiers, package declarations, and bare
calls (juxt_function_call). Vendor tree-sitter-groovy.wasm grammar.

Known limitation: fields declared without an explicit type (e.g.
`private repository`) are misparsed as method_declaration by the
tree-sitter grammar — typed fields (e.g. `private String name`) extract
correctly.
@ymstar

ymstar commented Jun 3, 2026

Copy link
Copy Markdown
Owner Author

Recreated against upstream

@ymstar ymstar closed this Jun 3, 2026
ymstar pushed a commit that referenced this pull request Oct 22, 2026
…ymchenry#720 (colbymchenry#748)

The per-word path fix (colbymchenry#745) brought the backend to parity but not above:
the project name still gave the lexically-matching stack a residual dir
match + an FTS class-name match, so a backend query that included the
project name still ranked the frontend at/above the backend.

Derive the project name from go.mod module / package.json name / repo dir,
and treat a query word matching it as non-discriminative: drop it from path
relevance and from codegraph_explore's PascalCase type-disambiguation bias
(reporter's suggestions #1/#2) — unless it's the only query word, so a bare
project-name search still scores.

Narrow by construction: the down-weighting fires ONLY when a query word
matches the derived project name (≥5 chars), so every query that doesn't
name the project is byte-identical. On the reporter's repro the backend
controllers now top a backend question that includes the project name;
queries without it, bare project-name queries, and normal symbol queries
are unchanged. Query-time only (no re-index).

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ymstar pushed a commit that referenced this pull request Oct 22, 2026
…nser frontend layer

codegraph_explore's file sort is primarily driven by Random-Walk-with-Restart
graph-centrality mass, seeded from the query's text matches. In a cross-layer
monorepo (an API server alongside a much larger, internally dense frontend that
mirrors the same domain words), that mass skews to the bigger layer — so a
backend service/handler that genuinely matches several query terms, even when
it's the #1 search hit, sorts below hits=0 frontend files and gets truncated out
of the response, and the agent reads it back.

Add a corroboration tier above the graph signal: a file that is BOTH an
entry/central file AND matched by >=2 distinct query terms is kept in. The
entry/central guard prevents an incidental multi-term file (a type/util file
that isn't the flow) from displacing a graph-central answer file — a blunt
hits-only tier regressed that case. Single-layer repos are unaffected. Gated by
CODEGRAPH_RANK_NO_MULTITERM=1.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ymstar pushed a commit that referenced this pull request Oct 22, 2026
…tier when they collide with real callables (colbymchenry#1252)

handleExplore's named-symbol seeding treats every identifier-shaped query
token as "a symbol the agent named" and grants its definition the
named-FIRST sort tier. Natural-language questions broke this assumption:
ordinary words exact-matched unrelated callables ("check" ->
WalCheckpointValve.check, "serve" -> query-worker serve, "initialize" ->
DatabaseConnection.initialize), and those collisions outranked — and within
the per-repo file budget fully displaced — the corroborated answer files,
forcing the agent back to Read/Grep. The >3-def single-pick fallback had
the same hole: on grpc, the colbymchenry#1064 flagship query "add a parameter to
NewClient" itself tiered balancerStateAggregator.add's file to slot #1.

Guard: a shape-precise token (camelCase, PascalCase, snake_case,
qualified) seeds unconditionally — it is an unambiguous symbol reference.
A bare lowercase word seeds only defs whose file another query token
co-names (that token is itself an exact symbol name defined in the same
file — the "check drain fire" sibling-bag shape), which an incidental
English-word collision never is. Applied by filtering cands ahead of both
branches so the overloaded-name fallback is covered too.

Validated per the retrieval playbook: deterministic probes on this repo
(collision queries fixed; sibling-bag and single-camelCase retained), and
baseline-vs-fixed probes on the colbymchenry#1064 repos — Alamofire and excalidraw
byte-identical, grpc improved (the add-collision file drops out and
clientconn.go + dialoptions.go lead). Full suite green.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
ymstar pushed a commit that referenced this pull request Oct 22, 2026
…pin, r default-routed (colbymchenry#1383)

R7b batch 4 #1 (docs/design/r-kernel-port-checklist.md is the authoritative
quirk list; survey + probe record therein). The lightest-shared-surface,
heaviest-hook port: languages/r.ts works entirely through the visitNode hook
(every type list empty except callTypes:['call']), so the walker is a file
node + a faithful hook transcription + the generic extractCall + pre-order
recursion — four shared machineries (value-refs, static-member reads, type
annotations, fn-ref capture) are dead by language gates and stay dead.

Grammar prep is the first true no-op of the arc: the crates.io tree-sitter-r
1.2.0 tarball ships parser.c AND scanner.c sha-identical to the r-lib v1.2.0
tag the vendored wasm was built from — crate pin only, no wasm change, no
bump gate; kernel-grammar-parity gains the r row (ABI 14, same-revision).

Preserved bug-for-bug (all probe-pinned): calls "return" on every return(x)
(named node in v1.2.0), the import quintet's silent dynamic-arg consumption
vs class/generic fall-through asymmetry, library(help = pkg) importing the
named arg, class-idiom variable suppression by callee name, chained/right-
assign/precedence-ghost gaps, env$fn body-leak-to-file, raw-text callees
verbatim (pkg::fn, obj$meth, "strfn" quotes kept, (handler) conversion),
duplicate same-(kind,name,line) ids, roxygen dropped entirely, UTF-16
columns/slices.

Gates: parity sweeps first-run 0-diff on AnomalyDetection/dplyr/ggplot2/
shiny (838 files; deferrals exactly 0/0/0/1 — the 1 is the moustache-
template pseudo-R file, both-arm) — kernel-parity.mjs gained lowercased-
extension matching so .R files sweep (matches detectLanguage routing);
full-init dumps byte-identical kernel-vs-wasm on dplyr/ggplot2/shiny;
kernel-r-parity suite (torture fixture + in-memory CRLF + BOM variants +
defer pin + kernel-arm quirk pins); full suite 2,638 green ×2 with
CODEGRAPH_KERNEL_EXPECT=1. DEFAULT_ROUTED += r (16 langs).

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
ymstar pushed a commit that referenced this pull request Oct 22, 2026
…ame-collision matches (CG-10, colbymchenry#1500)

Explore's per-file relevance awarded +50/+10/+3/+1 by match class and admitted
anything scoring >= 3. Neither half held up: the tier said HOW a symbol reached
us, never whether the match was evidence, and an absolute floor admits noise on
any repo where the top file scores 50+. Three scripts/agent-eval/*.mjs harnesses
took 63% of this repo's own "how does explore allocate its output budget" answer
on nothing but an unused `const explore` and a `const BUDGET`.

Four levers:

- KIND WEIGHT (RELEVANCE_KIND_WEIGHT): callables and types 1.0, members ~0.5,
  variable/constant/parameter 0.15-0.35. A weak-kind symbol with no usage edge
  anywhere in the graph (`contains` excluded — nesting is not usage) drops to
  0.08. Only weak kinds in the top two tiers pay for the DB probe; the subgraph's
  own edges answer most cases free. No measurable latency change (210 vs 211
  ms/call, n=12 interleaved).

- PERIPHERAL CAP: nodes >=2 hops from any match accumulate into a bucket capped
  at 5. Uncapped they added a flat +1 each, so a file grew more relevant by being
  bigger — parse-session.mjs reached 22 off one constant plus twelve unrelated
  symbols.

- RANK PENALTY: generated files x0.3, low-value x0.5, applied to the score AND
  the graph mass. Score alone would not have fixed colbymchenry#1500 — the generated CRUD
  carries MORE graph mass than the hand-written use-case, and graph mass outranks
  score in the comparator. Self-normalizing, never a hard exclusion.

- RELATIVE FLOOR: clamp(topScore * 0.2, 1, 10). Capped at one full-strength
  direct match so concentration elsewhere can never exclude one (without it a
  named-seed-heavy file pushed the floor to 21 and dropped a file the agent had
  named by class name). Backfills to 3 candidates when it would leave fewer, and
  drops the evidence requirement rather than return nothing at all.

excludeLowValueFiles was dead config — declared per tier, read nowhere; the
test/spec exclusion has been unconditional for a while. Removed. The real gap was
the detector: `isLowValue` anchored on a leading `/`, so a repo-ROOT `test/` dir
(express, cobra, most of npm and Go) never matched — express's routing question
spent 59% of its envelope on three test files. Anchored at `^` too, and the
filter now runs before the floor and judges "are there other candidates?" on the
whole gather.

Measured before/after on the same indexes (baseline bd86ad2):
- payroll-go fixture: generated 57.4% -> 23.5%; answer 25.6% -> 61.5%; cycle.go
  delivered 0 -> 38.9%. Generated ranks colbymchenry#3/colbymchenry#4, was #1/#2.
- self-query fixture: eval scripts 72% -> 0%; tools.ts ranks #1.
- express "route a request": 59% to test/* -> lib/application.js + lib/response.js
- cobra x3, codegraph "indexing pipeline": byte-identical (control)

Diagnostic gains a per-file penalty multiplier and NodeKind mix, so "why did this
file score X" is legible. Selection stages reordered to match the pipeline.

CG-6's gates flip from it.fails to live regressions except the byte-split ones,
which stay open for CG-12 (allocation still follows file size within the ranked
set).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ymstar pushed a commit that referenced this pull request Oct 22, 2026
…(CG-27)

CG-27 asked whether the >50%-of-file envelope drop should cover `function` /
`method`, so a `createFoo()` factory returning an object of closures stops
merging every closure inside it into one cluster. Measured on a hermetic
fixture, it should not, and the issue is closed as obsolete with CG-30 credited.

Two mechanisms already absorb the shape. shrinkCluster orders members by
(importance desc, size ASC) and refuses any member that overruns the cap once
something is kept, so a file-spanning member is only selected when it is the
sole member of the top importance tier — eight of nine query shapes never
selected it at all. When it IS selected, CG-30 windows it on whole lines, so
the file still delivers bounded, readable source (6 of 9 closure definitions
in that configuration).

Dropping the range instead SPLITS the file, and only the first-chosen cluster
may be shrunk: a trivial 7-line cluster won the density tiebreak and the
answer-bearing cluster was dropped whole — rank-#1 file 7,539 chars and 7 of 11
closures to 397 and none. Reaching the same intent more carefully (defer the
envelope MEMBER inside shrinkCluster, leaving clustering untouched) is noise:
69 vs 68 closure definitions across nine query shapes. Nothing shipped.

Adds the fixture, the probe, a standing gate on the outcome, and the record —
including a real defect the measurement exposed on the epic tip: django's
query.py leaves 8,212 of 10,135 unspent and drops a score-290 cluster to keep a
score-14 one. Filed separately.

No behaviour change, so no CHANGELOG entry.
ymstar pushed a commit that referenced this pull request Oct 22, 2026
CG-27 proposed adding function/method to ENVELOPE_KINDS so a factory closure
spanning most of its file stops merging every inner symbol into one cluster.
The issue required the ranking claim be measured before any fix. It was, on a
hermetic fixture built to make the pattern maximally visible, and it does not
hold — nothing shipped to src/.

The literal change is a large regression: dropping the enclosing range SPLITS
the file into a trivial cluster (a type alias plus a helper, span 7) and the
answer-bearing one (every closure, span 359). Cluster ranking breaks the equal
maxImportance tie on density, so the trivial cluster wins, is taken first, and
is the only one that may be shrunk; the answer-bearing cluster then does not fit
and is dropped whole. Rank #1 fell from 7,539 delivered chars to 397, and from
7 of 11 inner closures to 0. The enclosing range was holding the file together
as one cluster, inside which shrinkCluster already did the per-symbol ranking
the issue asked for.

A better mechanism reaching the same intent — deferring the envelope member
inside shrinkCluster, leaving clustering untouched — is noise: 69 vs 68 inner
definitions across nine query shapes, one better, one worse, seven unchanged.

The one configuration where the envelope IS selected (the factory as sole
top-tier member) is already absorbed by CG-30, which windows it on whole lines:
a contiguous readable head carrying 6 of 9 closures, bounded and never empty.

Kept: the fixture, the deterministic probe, and the measurement record.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ymstar pushed a commit that referenced this pull request Oct 22, 2026
A file that declares nothing but types and that nothing in the index depends
on — a hand-written ambient `.d.ts` of global shims, vendored typings, module
augmentation — cannot answer a flow question: no bodies, no call edges, no
behaviour, nothing typed by it. But the identifiers it declares are exactly the
generic ones a prose question uses (`Body`, `Message`, `ImageMetadata`,
`ReadableStream`), so on term overlap it out-scored the implementation. Measured
on the new fixture: rank #1 and 51% of delivered source, with the flow's own
entry file pushed out of the response entirely.

Measured first, per the issue: the Wrangler `worker-configuration.d.ts` that
opened this is already handled by CG-25's banner detection, worth 15-46 points
of envelope share across four flow queries. CG-25 credited; only the un-bannered
case needed anything.

`rankPenalty` now multiplies score and graph mass by 0.5 for such files, taken
as the STRONGER of it and the generated penalty rather than multiplied — one
property two signals see must not be charged twice. Detection is structural, not
by extension, and four conditions deep. Two of them were forced by measurement:
requiring every symbol to be type-level takes the corpus flag rate from 1-18%
(which swept in Kotlin sealed classes, Rust mod.rs re-exports and django's
locale tables) down to 0-4%; requiring that nothing depends on the file
separates an ambient shim from a working types module, and without it the rule
demoted displacement-ts's pipeline `types.ts` and broke the CG-31 gate.

A query that NAMES a declared type is exempt, so a question about a type still
reaches its declaration at full weight. Precise tokens only, so "…the file
body…" cannot exempt a `Body` interface it never meant to name; this needs its
own set because `namedSeedIds` is callable-only and a type never becomes one.

Regression evidence in docs/benchmarks/explore-declaration-only-cg28.md:
6-repo envelope sweep byte-identical against a clean baseline build, zero
ambient files reach the candidate set on VS Code across five queries, corpus
flag rate 0-0.74%, both allocation fixtures PASS, full suite 2,978 green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ymstar pushed a commit that referenced this pull request Oct 22, 2026
…CG-28)

Both halves of the issue were measured on a hermetic fixture of four
declaration-shaped files varying on banner and depended-on-ness.

CG-25 already handles the motivating file: the Wrangler worker-configuration.d.ts
that opened this issue is demoted by the generated penalty alone, worth 15-46
points of envelope share across four flow queries. No new mechanism for it.

The narrower gap is real. A declaration file with NO banner carried pen 1.00,
took rank #1 and 51% of delivered source on a prose flow query, and displaced
the flow's own entry file out of the response entirely.

The rule is deliberately narrow, and both conditions were derived by survey
rather than guessed. 'Declares no callable and calls nothing' flags 1.1-18.0%
of files across the corpus and catches real source — okhttp's SocketPolicy.kt,
tokio/src/runtime/mod.rs, Alamofire's umbrella file, django's locale format
tables. Requiring every symbol to be type-level drops that to 0-4%. The
'nothing depends on it' condition was added after the broader version demoted a
pure-interface file with 13 inbound imports and broke the CG-31 displacement
gate — a different invariant entirely.

Does NOT stack with the generated penalty: rankPenalty takes Math.min of the
two, so a file that is both takes the stronger, never the product. A query that
NAMES a declaration symbol exempts its file entirely, so asking about a type
still reaches it at full weight.

Six-repo envelope is byte-identical to the pre-change tip — the rule does not
fire on any benchmark repo, consistent with the 0-4% survey.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ymstar pushed a commit that referenced this pull request Oct 22, 2026
…ts (CG-38)

`codegraph_explore` never returned `queueMessage` (L1087) or
`flushQueuedMessages` (L1102) from a 1,414-line file, on a symbol bag or a
prose question, even with that file at rank #1 holding 67% of the envelope —
the agent got a same-stem `QueuedMessage` interface at L70 and had to Read the
file for the functions it had named. Pre-existing at every build including
pre-epic (controlled bisect, index held fixed).

Two independent causes:

1. `buildFlowFromNamedSymbols` returns the Flow prose AND the set of node ids
   the agent named — and the latter is the whole guarantee, since it injects a
   named def into its file's cluster ranges at importance 9. Its bail-outs
   returned EMPTY, zeroing the identity whenever there was nothing to PRINT.
   Two sibling closures that never call each other produce no chain, no synth
   hop and no boundary, so both defs lost importance 9 and the file rendered
   from its head. `identityOnly()` now separates the two, gated on
   shape-precise tokens so a prose word that exact-matches a callable cannot
   promote itself.

2. The ceiling trim filled in SOURCE order, so an over-ceiling render always
   dropped the END of a large file first. The shrink HAD kept both symbols
   (1022-1121); the trim cut back to 839. `windowToCeiling` now takes the
   spine call site plus every importance>=9 member as focus lines, tries the
   full ceiling first, and splits the held-back reserve evenly with
   carry-forward — greedy-in-source-order reproduced the bug one level down.

The shrink's loose size estimate is left alone deliberately, and the comment
now says why: making it exact was built and measured WORSE (it stops at the
last member that fits whole and the released bytes carry forward to
lower-ranked files, costing payroll-go's `s.store.Upsert`). `bound()` clamps to
the ceiling anyway, so the slack costs no bytes; it just must not pick the
survivors, which is what the trim now handles.

The measurement gap this closes: every existing probe is aggregate — envelope
share, per-file spend, source totals, file counts — and all are green on a
response that returns 25K from the right file and omits the named function.
`probe-named-symbol.mjs` checks the definition LINE against the response's
rendered lines, per symbol.

Suite envelope byte-identical to main on all six repos; probe-allocation 4/4,
no starvation flags; 180 files / 2,997 tests green. Fixture: 7/7 fail on main,
7/7 pass here, deterministic over 4 runs per arm.
ymstar pushed a commit that referenced this pull request Oct 22, 2026
…r deep files to wasm (colbymchenry#1581) (colbymchenry#1600)

Fixes colbymchenry#1581.

## What was wrong

`codegraph init` / `codegraph index` died with `Segmentation fault` — the whole CLI
process, not a parse worker — on a C/C++ file with very deep brace nesting (llvm's
`clang/test/Parser/parser_overflow.c`, 16,384 nested `{`). The reporter's diagnosis is
exactly right: tree-sitter's parser is iterative, so the file parses fine, and then the
native kernel's **recursive walker** (`visit_node` → `visit_for_calls_and_structure` → …,
one frame per AST level) overflowed the thread's stack. A native overflow can't be caught
the way a wasm abort can, and a parse worker is a thread of the `codegraph` process, so
the SIGSEGV took the entire indexer down — no message, no per-file fallback, no partial
index.

Two things made "just give the worker a bigger stack" the wrong fix:

- it only moves the cliff — reproduced here: the reporter's 16,384-deep file kills a
  default 4 MiB worker (rc=132 on macOS / 139 on Linux), and a 100k-deep file kills the
  8 MiB **main** thread too;
- the walkers are shared by every kernel-routed language (20 of them), and each has
  several recursion points with different frame sizes, so no single stack size is a
  provable bound.

Meanwhile the wasm path already handles this shape gracefully: its JS walker catches its
own `RangeError` per file and stores a partial result with a `parse_error`. The kernel
just needed a way to get there instead of dying.

## What this does

**The kernel guards its own recursion against the calling thread's real stack bounds and
defers a too-deep file to wasm** — the same `defer:` routing signal it already uses for
files with parse errors, which `src/extraction/kernel/index.ts` treats as "take the wasm
path for this file", silently.

- `codegraph-kernel/src/stack.rs`: per-thread stack bounds from the OS, computed once per
  thread and cached — glibc/musl `pthread_getattr_np` + `pthread_attr_getstack`, macOS
  `pthread_get_stackaddr_np` + `pthread_get_stacksize_np`, Win32
  `GetCurrentThreadStackLimits` (a hand-declared `kernel32` extern; no `windows-sys`).
  `exhausted()` is one thread-local load and one compare: true once the stack pointer is
  within a 256 KiB red zone of the limit, and it latches a flag. Where the OS can't report
  bounds it falls back to a fixed 1 MiB descent budget measured from the entry stack
  pointer — safe on anything from Node's 4 MiB worker default up. So the guard is exact on
  the 4 MiB worker, the 8 MiB main thread, and any `resourceLimits.stackSizeMb` alike.
- `stack_guard!()` (defined in `lib.rs`) is the first statement of every recursive walker
  function — all **150** self-recursive or on-cycle functions across the 15 walker modules,
  found by script (every cycle in the call graph, not just direct self-calls). It returns
  `Default::default()` (`()`, `false`, `None`, `""`) so an exhausted walk simply stops
  descending; a hook returning `false` sends its caller down the generic child walk, whose
  own guard returns at once.
- `extract_file` runs the whole walk under `stack::run_guarded`: if the flag is set
  afterwards the (truncated) result is discarded and replaced by
  `defer: nesting too deep for the native walker — wasm recovery handles it`.
- `parse-pool.ts`: a comment at `new Worker(scriptPath)` records why there is deliberately
  no `resourceLimits.stackSizeMb` bump.
- No new crates beyond `libc` as a direct unix dependency (already in `Cargo.lock`
  transitively). No wire/ABI change.

Net effect for the reporter's repo: `deep.c` goes to the wasm path, lands as
`function foo` plus a recorded parse warning, and the other 31,607 files index normally.
`CODEGRAPH_KERNEL=0` and the `exclude` workaround are no longer needed.

## Tests

**Rust unit tests** (`cargo test`, 21 passed — 7 new in `stack.rs`): the walkers for
C, C++, Rust, TypeScript and Python are driven on a **1 MiB** thread (a quarter of Node's
worker default) with 30k-deep nesting and must return `defer:` instead of crashing;
shallow files are untouched; the latch resets between runs; the OS bounds are sane on the
main thread and describe a small thread's own stack.

**`__tests__/kernel-deep-nesting.test.ts`** (new, 8 tests — skips without a staged `.node`,
fails under `CODEGRAPH_KERNEL_EXPECT=1` if the kernel is missing, like the other kernel
suites):
- every default-routed language (all 20) survives a 60k-deep expression on the main thread
  — clean result or the wasm fallback's partial result, never a crash;
- the reporter's exact 16,384-brace C file is indexed (partial) on the main thread;
- 200-deep expressions in every language still take the kernel path clean (the guard never
  trips on normal code);
- inside a **default-sized 4 MiB `worker_threads` Worker** through `dist/`: the reporter's
  `deep.c` and a 60k-deep expression in every language come back `deferred` with exit 0,
  and a normal file still extracts natively;
- end-to-end through the built CLI: `codegraph init` on a repo holding `deep.c` + `ok.c`
  exits 0 and records both files, with both functions.

**Existing kernel suites**: all 15 (`kernel-*-parity`, `kernel-scaffold`,
`kernel-retry-materialize`, `kernel-grammar-parity`) pass unchanged, 147 tests — the guard
never fires on the parity fixtures.

**Reporter's probes** (`one.js` from the issue, default 4 MiB worker, this build):
`deep.c` → `deferred`, exitCode=0 (was rc=132/139); `deep100k.c` → `deferred`, exitCode=0.
Main thread: `deep.c` / `deep100k.c` → wasm partial with
`Parse error: Maximum call stack size exceeded`; a 6,000-term binary expression and a
3,000-branch `else if` chain stay on the kernel path with clean results.

**Perf** (same `dist/`, only the `.node` swapped via `CODEGRAPH_KERNEL_PATH`; interleaved
main/new ×3, `codegraph init`, macOS arm64):

| repo | main (median) | guarded (median) | nodes / edges |
|---|---|---|---|
| express (141 files) | 0.60 s (0.58–0.65) | 0.61 s (0.58–0.61) | 1,084 / identical |
| redis (786 C/H files) | 4.44 s (4.39–4.66) | 4.49 s (4.41–4.70) | 19,942 / 76,446 identical |

Within run-to-run noise, as expected for one TLS load + compare per recursion entry.

**Linux (Docker, `node:22-bookworm`, kernel built in-container, `docker run --rm --init`)** —
the reporter's platform and the glibc `pthread_getattr_np` bounds path:

```
=== platform ===
Linux efe3cc86947b 6.12.54-linuxkit #1 SMP Tue Nov  4 21:21:47 UTC 2025 aarch64 GNU/Linux
v22.22.3
-rwxr-xr-x 1 root root 35332288 Aug 22 18:02 codegraph-kernel/prebuilds/linux-arm64/codegraph-kernel.node

=== reporter repro (issue colbymchenry#1581): 16,384-brace deep.c, codegraph init ===
│
└  Done

init exit code: 0
  file: deep.c
  file: deep100k.c
  file: ok.c
  function: add
  function: bar
  function: foo

=== worker probe: kernel raw extract in a default 4 MiB worker ===
deep.c: deferred
deep.c: worker exitCode=0
deep100k.c: deferred
deep100k.c: worker exitCode=0
ok.c: kernel nodes=2
ok.c: worker exitCode=0

=== cargo test stack:: (glibc pthread_getattr_np bounds path) ===
test stack::tests::os_bounds_are_sane_on_this_platform ... ok
test stack::tests::small_stack_reports_its_own_bounds ... ok
test stack::tests::normal_files_are_untouched_by_the_guard ... ok
test stack::tests::deep_braces_c_defer_instead_of_crashing ... ok
test stack::tests::latch_resets_between_runs ... ok
test stack::tests::deep_parens_cpp_rust_ts_python_defer_instead_of_crashing ... ok
test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 15 filtered out; finished in 0.23s

=== vitest: kernel-deep-nesting + kernel-scaffold ===
✓ __tests__/kernel-scaffold.test.ts (10 tests) 30ms
✓ __tests__/kernel-deep-nesting.test.ts (8 tests) 36989ms
Test Files  2 passed (2)
Tests  18 passed (18)
```

(The pre-fix crash was reproduced on macOS — rc=132 in a default worker, rc=139 on the main thread at 100k depth — not re-run inside this container; the reporter's Linux x86_64 trace is the SIGSEGV form of the same overflow.)

**Windows (Parallels ARM64 VM, MSVC 14.44, `cargo 1.97`, kernel built on the VM,
`GetCurrentThreadStackLimits` path)**:

```
head: cbf8485 fix(kernel): guard the native walkers against stack overflow and defer deep files to wasm (colbymchenry#1581)
=== cargo build --release (win32-arm64) ===
    Finished `release` profile [optimized] target(s) in 2m 04s
staged: 35086848 bytes
=== cargo test (stack guard unit tests) ===
test stack::tests::normal_files_are_untouched_by_the_guard ... ok
test stack::tests::os_bounds_are_sane_on_this_platform ... ok
test stack::tests::small_stack_reports_its_own_bounds ... ok
test stack::tests::deep_braces_c_defer_instead_of_crashing ... ok
test stack::tests::latch_resets_between_runs ... ok
test stack::tests::deep_parens_cpp_rust_ts_python_defer_instead_of_crashing ... ok
test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 15 filtered out; finished in 0.49s
=== reporter repro: codegraph init on a 16,384-brace deep.c ===
└  Done
init exit code: 0
=== vitest: deep-nesting + scaffold (CODEGRAPH_KERNEL_EXPECT=1) ===
✓ __tests__/kernel-scaffold.test.ts (10 tests) 55ms
✓ __tests__/kernel-deep-nesting.test.ts (8 tests) 67239ms
   ✓ every default-routed language survives a 60k-deep expression on the main thread 52801ms
   ✓ inside a default-sized (4 MiB) parse worker, through dist/ > defers a 60k-deep expression in every default-routed language 13050ms
   ✓ end-to-end: codegraph init on a repo holding the deep file > exits 0 and records deep.c alongside the normal files 936ms
Test Files  2 passed (2)
Tests  18 passed (18)
```

(The end-to-end test is what reads the Windows index back through `node:sqlite` — `files` = `deep.c`, `ok.c`; functions `add`, `foo`.)

Full `npm test` on this branch (macOS arm64, kernel staged): **190 files passed, 3,185 tests passed, 10 skipped, 0 failed.**

Clippy note: `cargo clippy` on the current toolchain (1.92) reports 18 pre-existing lints
(`manual_contains`, `unnecessary_to_owned`, …) in walker code this PR only touched by
inserting guard lines; none are in `stack.rs`/`lib.rs`. Left alone to keep the diff
reviewable.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01LxZj6W6Y1SHXwvpT3uwJpK
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add support for Groovy language

1 participant